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WHAT’S MY METHOD? 

USABILITY: Game evaluation 
through the direct observation 
of player success and failure. 
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DIRECT OBSERVATION 




PRIMARY BEHAVIOR 

What’s the player doing inside the game world? 
Can the player perform the task? 

Does the player succeed or fail? 

SECONDARY BEHAVIOR 

What happens outside the game world? 

© body language, facial expressions 
© controller interaction 
© talking aloud, interview responses 

Helps us interpret the primary behavior. 
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STEP 1: DEFINE GOALS 



® INTERFACE 

correctly define key interface elements 
use them appropriately during game play 
TASK: ID Gang Notoriety Meter 

® CRITICAL SKILLS 

able to attempt skill 
improve over time 
TASKS: Driving, Shooting 

® KEY CONCEPTS 

describe function and value 
retain knowledge 
TASK: Respect 
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FIND PLAYERS 
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SAMPLE 

recruited locally 

own or play games on XBox 360 
some familiarity with GTA 

4-6 PLAYERS 

caveat: know your limitations 

usability measures SUCCESS and FAILURE 

NOT preferences or opinions 



Jakob Neilsen’s Alertbox, March 19, 2000 

“Why You Only Need to Test With 5 Users” 
http://www.useit.com/alertbox/20000319.html 
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BUILD YOUR LAB 
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COST 

hi-budget or lo-budget 

temporary or permanent 

just need a place to sit, pen & paper 

PARTICIPANTS 

player 

moderator 

observers 


® VOLITION LAB 

participant room (comfortable) 
observation room (quiet) 

1-way mirror + video feed 
DV capture (controller, face, screen) 
reception area (hall) 
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STEP 4: RUN THE TEST 



® SCRIPT 

Part 1: Free Play 
Part 2: Questions 
Part 3: Directed Play 
Part 4: Interview 

© PROTOCOL 

player — think aloud 
you — shut up & observe 
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STEP 5: WRITE IT DOWN 
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Aiming reticle moves erratically around the I 
screen. I 

Vehicle stuck. Player can’t find 
reverse. 


Player misses turns when ■ = 

navigating by GPS. 

Player fails to ID respect 
_meter. 
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STEP 6: TAKE ACTION 


Issue 

Action 

Aiming reticle moves erratically 
around the screen. 

Implement aiming assist to help 
reticle stick to target. 

Vehicle stuck. Player can’t find 
reverse. 

Display basic driving controls 
when player first enters vehicle. 

Player misses turn when 
navigating by GPS. 

Tell GPS to generate new route 
if player deviates from old route. 

Player fails to ID respect meter 

Require player confirmation 
when tutorial text is displayed. 
Fill respect meter on activity 
completion screen. 
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STEP 7: MORE TESTS! 




Test 

Focus 

Date 

Subjects 

Time 

1 

Controls / Interface 

OCT 05 

10 

45 minutes 

2 

Tutorial: Interface / Skills / Concepts 

JAN 06 

5 

100 minutes 

3 

Prologue 

MAY 06 

5 

180 minutes 

4 

Multiplayer 

JUN 06 

4 

100 minutes 

5 

Balancing 

MAY 06 

3x3 

40 hours 

6 

Benchmarking 

JUL06 

3x3 

40 hours 
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USABILITY: PROS/CONS 
(+) 

you can do it yourself 
it’s cheap and fast 
you get actionable data 
designers observe players directly 



not a reliable measure of preferences or opinions 
no fancy headgear 
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